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Abstract 

A  core  tenet  of  spiral  development  and  evolutionary  acquisition  concepts  is  the  ability  to 
insert  new  technologies  into  an  existing  system  on  an  as-needed  basis,  as  they  mature,  in  order 
to  minimize  risk  and  maximize  affordability.  Through  this  continual  rolling  in  of  evolving 
components  the  system  continues  to  offer  more  advanced  capability.  This  creates  an  elaborate 
tradeoff  scenario  in  which  dissimilar  attributes  must  be  examined,  weighted,  and  analyzed  for 
best  value  and  applicability  to  user  needs  and  requirements,  including  timing.  A  further 
complication  for  system-of-systems  is  added  by  the  need  to  give  equal  consideration  and 
analysis  to  each  technology’s  ability  to  be  integrated  with  existing  system  components  in  a 
functional  architecture  as  well  as  any  impact  on  current  and  interconnected  capabilities.  To 
address  this  need  for  a  multi-attribute  decision-making  tool,  NAVSEA  and  the  Northrop 
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Grumman  Corporation,  along  with  partners  at  the  Stevens  Institute  of  Technology  and 
SPAWARSYSCEN  Pacific,  have  collaborated  to  define  a  holistic  approach  for  evaluating 
technology  insertion  options  from  a  complex  system-of-systems  integration  perspective. 
Through  this  paper,  we  will  discuss  the  tool’s  potential  to  aid  the  decision-maker  in  the  selection 
of  best  value  technologies  and  its  potential  utility  as  a  critical  piece  of  the  unified  system 
engineering  and  acquisition  process. 

Overview  of  the  Current  System-of-systems  (SoS)  Acquisition 
Environment 

Current  Department  of  Defense  (DoD)  acquisition  activities  continue  to  push  the 
integration  envelope  with  the  development  of  larger  and  more  complex  systems-of-systems.  In 
many  ways,  this  development  paradigm  invalidates  many  of  the  models,  historical  databases, 
and  even  engineering  expertise  that  have  been  used  for  decades  in  the  development  of  stand¬ 
alone  systems.  Similarly,  the  system-of-systems  revolution  has  made  management  of 
acquisition  programs  more  difficult,  as  keeping  accurate  and  current  control  of  the  countless 
moving  parts  of  systems  development  is  nearly  impossible  due  to  the  exponential  growth  of 
technologies  and  integrations  being  incorporated  under  a  common  system-of-systems  banner. 
This  fact  necessitates  the  development  of  a  new  set  of  tools  and  best  practices  in  order  to 
manage  the  many  unique  aspects  of  development  associated  with  system-of-systems. 

Unique  SoS  monitoring,  assessment,  and  management  needs 

Nowhere  is  the  need  for  enhanced  monitoring  capabilities  more  visible  than  in  the  SoS 
development  maturity.  For  the  better  part  of  two  decades,  the  Technology  Readiness  Level 
(TRL)  methodology  has  been  key  in  gauging  the  current  maturity  status  of  a  given  piece  of 
technology  within  the  DoD.  By  monitoring  capability  development  from  concept  definition 
through  operations  and  support  using  the  TRL  series  of  nine  levels  of  maturity,  the  readiness  of 
a  technology  for  integration  into  a  system  has  been  adjudicated.  In  countless  development 
efforts,  TRL  has  been  key  in  indicating  progress  and  has  aided  dramatically  in  keeping 
numerous  programs  on  track.  Indeed,  it  has  been  incorporated  as  a  critical  tollgate  criterion  in 
the  Defense  Acquisition  Milestone  process.  However,  when  TRL  is  applied  to  components 
within  a  system-of-systems,  the  model  of  using  individual  technology  maturity  as  a  measure  of 
readiness  to  integrate  into  system  development  quickly  breaks  down.  TRLs  do  not  account  for 
integration  maturity  or  the  complexity  of  bringing  together  any  number  of  independent 
technologies  to  function  as  a  common  system.  Similar  problems  also  become  apparent  with 
many  other  technology  development  tools  when  applied  in  a  system-of-systems  context.  This 
lack  of  adequate  system-of-systems  level  development  monitoring  tools  and  methodologies  has 
resulted  in  a  rash  of  complex  development  and  acquisition  projects  going  astray.  The  General 
Accounting  Office  (GAO)  noted  that  a  lack  of  insight  into  the  technical  maturity  of  complex 
systems  during  development  has  contributed  to  an  environment  of  significant  cost  overruns, 
schedules  slips  leading  to  program  delays,  canceled  acquisition  efforts,  and  reduced  system 
performance  at  fielding  (2006).  In  case  after  case,  failure  is  not  commonly  found  at  the 
technology  development  level,  but  rather  at  the  point  of  combination  of  two  or  more  elements. 

In  order  to  mitigate  this  identified  risk,  PMS  420,  the  Littoral  Combat  Ship  Mission 
Module  Program  Office  has  previously  implemented  an  emerging  concept  known  as  the  System 
Readiness  Level  (SRL).  By  pairing  the  traditional  TRL  scale  with  a  new  series  of  criteria  known 
as  the  Integration  Readiness  Level  (IRL),  a  more  complete  look  at  true  system  maturity  can  be 
obtained  (Sauser,  Ramirez-Marques,  Magnaye  &Tan,  2008).  Under  this  methodology  the 
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readiness  of  each  technology  is  still  considered,  but  instead  of  being  a  stand-alone  metric  for 
determining  readiness  for  incorporation,  it  is  analyzed  in  concert  with  both  its  integration 
requirements  and  the  maturity  of  other  technologies  with  which  it  interfaces.  The  SRL 
methodology  has  been  highly  successful  on  the  program  and  has  paid  dividends  in  terms  of 
both  increasing  decision-maker  visibility  into  true  system  status  and  allowing  for  pre-emptive 
actions  to  be  taken  to  mitigate  potential  developmental  issues.  PMS  420  is  looking  to  expand 
upon  the  foundation  of  system  readiness  monitoring  laid  by  the  SRL  concept  and  expand  it  to 
new  uses  in  both  guiding  technology  selection,  insertion  and  tradeoffs  as  well  as  for  use  in  cost 
modeling  in  order  to  understand  the  impacts  of  implementing  technology  options. 


Initial  Step — Understanding  the  Current  System 

A  core  tenet  of  systems  engineering  is  to  fully  understand  and  capture  the  architectures 
of  the  system  being  developed.  This  includes  obtaining  a  comprehensive  background  on  the 
individual  components  and  technologies  as  well  as  the  ramifications  of  their  proposed 
integration  or  networking.  In  case  after  case,  however,  it  can  be  seen  that  programs  have 
entered  acquisition  with  incomplete  or  inaccurate  mappings  of  these  most  basic  of 
considerations.  The  SRL  concept  enforces  a  degree  of  accountability  by  requiring  consideration 
be  given  to  mapping  of  an  architecture  and  the  maturity  of  the  individual  pieces  being  brought 
together  prior  to  action  being  taken. 

Upon  the  start  of  the  Mission  Module  Program,  the  ability  to  pull  together  and  assess  a 
wide  variety  of  components  at  numerous  developmental  maturity  states  was  a  necessity.  As  the 
provider  of  a  set  of  interchangeable  and  standards-based  mission  modules  for  the  Littoral 
Combat  Ship,  PMS  420  was  tasked  to  leverage  a  considerable  amount  of  technology  from 
existing  programs  of  record  in  a  “come  as  you  are”  development  effort.  This  was  done  to 
facilitate  quick  fielding  of  desperately  needed  capability  in  the  areas  of  mine  countermeasures, 
anti-submarine  warfare,  and  surface  warfare.  This  rapid  development  environment  resulted  in 
the  selection  of  technologies  from  a  considerable  mix  of  existing  GOTS  and  COTS  products 
along  with  new  development  efforts.  Initially,  integration  of  the  capabilities  was  not  an  objective, 
but  it  rapidly  became  a  necessity.  Thus,  the  Mission  Module  Program  needed  to  track  not  only 
the  widely  varying  maturity  status  of  the  technologies  but  also  the  various  integrations  activities 
between  them  as  a  critical  function  of  management  control.  The  SRL  methodology  was  used  to 
capture  this  complex  and  diverse  acquisition  effort  and  provide  snapshots  of  program  status, 
technology  maturity  and  integration  risks  and  issues. 

SRL  Concept 

Since  being  introduced  by  NASA  in  the  early  1990’s,  the  TRL  has  steady  gained 
widespread  acceptance  as  a  powerful  tool  for  its  use  in  assessing  technology  maturity.  In  order 
to  build  upon  the  successes  of  this  tool,  the  SRL  methodology  leverages  the  traditional  TRL 
scale  as  its  core  for  assessing  the  maturity  of  individual  technologies  within  the  system-of- 
systems.  The  TRL  scale  is  then  paired  with  a  parallel  evaluation  scale,  known  as  the  IRL,  to 
capture  integration  status  between  individual  components.  Much  like  the  TRL,  IRL  is  a  nine- 
level  scale  capturing  evolving  levels  of  maturity  for  two  components.  Though  it  is  natural  for 
integration  to  slightly  lag  technology  maturity,  the  IRL  closes  follows  the  TRL  scale  as  it  tracks 
integration  maturity  development  from  concept  to  operational  system.  Table  1  provides  a  high- 
level  definition  of  the  IRLs  (Gove,  Sauser  &  Ramirez-Marquez,  2009).  The  development  of  SRL 
has  been  led  by  a  joint  team  of  researchers  from  the  Stevens  Institute  of  Technology,  Northrop 
Grumman  Corporation,  SPAWARSYSCEN  Pacific,  and  PMS  420.  Full  reports  on  the  creation 
and  validation  of  the  SRL  concept  have  been  provided  in  a  series  of  academic  papers  and 
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presentations.  The  concept  has  powerfully  displayed  insight  into  complex  system-of-systems 
development  maturity. 


Table  1.  Integration  Readiness  Level  Definitions 


IRL 

Definition 

9 

Integration  is  Mission  Proven  through  successful  mission  operations. 

8 

Actual  integration  completed  and  Mission  Qualified  through  test  and  demonstration,  in  the 
system  environment. 

7 

The  integration  of  technologies  has  been  Verified  and  Validated  with  sufficient  detail  to  be 
actionable. 

6 

The  integrating  technologies  can  Accept,  Translate,  and  Structure  Information  for  its  intended 
application. 

5 

There  is  sufficient  Control  between  technologies  necessary  to  establish,  manage,  and 
terminate  the  integration. 

4 

There  is  sufficient  detail  in  the  Quality  and  Assurance  of  the  integration  between  technologies. 

3 

There  is  Compatibility  (i.e.,  common  language)  between  technologies  to  orderly  and  efficiently 
integrate  and  interact. 

2 

There  is  some  level  of  specificity  to  characterize  the  Interaction  (i.e.,  ability  to  influence) 
between  technologies  through  their  interface. 

1 

An  Interface  between  technologies  has  been  identified  with  sufficient  detail  to  allow 
characterization  of  the  relationship. 

One  of  the  most  commonly  recognized  shortcomings  of  readiness  scales  is  their 
inherent  subjectivity  in  evaluation  due  to  the  fact  that  ratings  are  often  determined  by  individual 
assessors  using  qualitative  data.  The  SRL  methodology  implements  an  analytical  approach  to 
help  to  mitigate  some  of  these  concerns.  Steps  have  also  been  taken  to  enhance  the  quality  of 
the  TRL  and  IRL  evaluations  that  feed  it  by  creating  detailed  evaluation  criteria  to  minimize  the 
opportunity  for  subjective  interpretation.  In  order  to  assess  the  SRL  of  a  given  system,  each 
component  of  the  end  capability  (i.e.,  single  system  or  a  system-of-systems)  is  rated  with 
respect  to  its  TRL  or  IRL.  These  are  then  combined  into  a  TRL  and  IRL  matrix  as  shown  in 
Figure  1 . 
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Figure  1.  SRL  Calculation 


After  normalizing,  the  matrices  are  multiplied  forming  a  SRL  vector.  This  vector  is  known 
as  “component  SRL”  and  represents  each  of  the  technologies  within  the  system,  considering  all 
its  integrations.  These  individual  technology  SRL  assessments  provide  powerful  insight  into  the 
maturity  and  integration  status  of  each  technology  from  an  end  capability  perspective. 
Additionally,  they  offer  an  indication  of  which  elements  are  lagging  and  which  are  ahead  in 
development  within  the  system.  The  individual  SRL  scales  can  also  be  averaged  to  provide  an 
overall  SRL  rating  for  total  system-of-systems  capability.  This  single  score  is  known  as  a 
“composite  SRL”  and  functions  as  a  roll-up  of  the  individual  component  SRLs,  providing  insight 
into  the  level  of  maturity  and  integration  of  the  total  capability.  It  is  important  to  note  that  each 
assessment  is  critical  to  assessing  the  state  of  the  overall  system.  Simply  examining  an  overall 
SRL  score  does  little  without  understanding  the  impact  of  maturity  and  integration  status  of  each 
individual  component.  In  a  large  system,  a  single  immature  piece  could  easily  be  masked  in  a 
composite  SRL  number  but  such  an  act  would  be  evident  when  assessed  at  a  component  SRL 
level.  Likewise,  composite  SRL  provides  a  good  indication  of  overall  development  status  and 
the  magnitude  of  remaining  work,  but  it  could  mask  the  inability  of  the  system  to  function  due  to 
a  single  capability  with  low  levels  of  maturity  or  integration  at  a  key  juncture  within  the  overall 
system-of-systems. 

An  overview  of  the  overall  SRL  assessment  is  provided  in  Figure  2.  It  is  important  to 
note  that  the  actual  SRL  calculation  is  one  part  of  a  larger  exercise  in  defining  and  evaluating 
the  overall  system  architecture. 


DEFENSE  ACQUISITION  IN  TRANSITION 


- 135- 


Step  1 :  Identify  hardware  and 
software  components 


Include  all  technologies  that  make-up 
the  overall  system 


Step  2:  Define  network  diagram 
for  systems 


Emphasis  is  on  the  proper  depiction  of 
hardware  and  software  integration 
between  the  components 


Initial  Architecture  Definition  and  Setup 


step  4:  Apply  detailed  TRL  and  IRL 
evaluation  criteria  to  components  and 
integrations 


Step  5:  Calculate  individual  and 
composite  SRLs 


Step  6:  Document  status  via  roll¬ 
up  charts 


Checklist  style  evaluation  allows  for  the 
ability  to  ‘‘take-credit’’  for  steps  that  have 
taken  place  beyond  the  current  readiness 
level 


Input  TRL  and  IRL  evaluations  into 
algorithm  to  compute  an 
assessment  of  overall  system  status 
via  SRLs 


Populate  reporting  chart  templates 
with  evaluation  and  calculation 
outcomes  to  highlight  both  current 
status  and  performance  over  time 


Iterative  SME  Evaluation  Throughout  Development  Cycle 


Figure  2.  SRL  Analysis  Process 


SRL  Applications  in  Decision-making 

The  SRL  methodology  has  proven  to  be  of  tremendous  use  and  utility  in  evaluating 
current  status  and  then  providing  the  needed  insight  in  order  to  determine  the  appropriate 
course  of  action.  In  a  complex  SoS  environment,  it  is  not  always  immediately  clear  where 
resources  should  be  applied  for  most  efficient  application  in  order  to  maximize  system  maturity 
and  minimize  risk.  By  allowing  for  trade-off  analysis  and  “what-if  scenarios,  the  SRL  lends  itself 
to  analysis  of  overall  system  impact,  which  allows  for  a  wide  variety  of  combinations  to  be  tried 
before  dollars  are  ever  spent.  In  this  way,  a  new  technology  or  development  option  can  be 
inserted  into  the  architecture  and  its  impact  on  overall  maturity  analyzed.  An  example  of  this 
trade-off  analysis  as  applied  on  the  Mission  Module  program  can  be  found  in  Figures  3  and  4. 

The  figures  represent  the  architecture  of  one  SoS  on  the  Mission  Modules  Program. 
Technologies  are  located  in  blue  boxes  while  the  lines  between  them  denote  integrations.  The 
assessed  TRL  and  IRL  ratings  can  be  found  in  teal  and  purple  boxes,  respectively.  The 
component  SRL  ratings  are  denoted  by  small  triangles  across  the  top  of  the  development  scale 
at  the  bottom  of  the  figure,  while  the  overall  composite  SRL  readiness  number  appears 
underneath  the  scale.  The  overall  system  maturity  is  exceeding  the  scheduled  development 
position,  which  is  indicated  by  the  dotted  red  line  and  is  determined  via  an  SRL  to  program 
Integrated  Master  Schedule  mapping.  However,  one  of  the  system  technologies,  the  MVCS 
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(RMMV),  has  fallen  significantly  behind  in  planned  development.  This  technology  serves  as  a 
vital  communication  link  in  the  command-and-control  chain  between  the  ship  and  an  unmanned 
vehicle.  With  a  risk  item  in  the  system  development  identified,  mitigation  options  were 
generated,  including  increasing  development  resources  or  inserting  an  alternate  technology. 
After  examining  projected  performance,  cost,  and  schedule  numbers  for  each  option  as  well  the 
impact  on  overall  system  maturity,  it  was  determined  that  a  more  mature  technology  would  be 
inserted  for  initial  spirals.  While  this  option  offered  less  capability  in  the  near-term,  it  ensured 
performance  requirements  were  met  while  enhancing  overall  system  maturity  and  reducing  risk. 
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Figure  3.  Initial  System  Readiness  with  Lagging  Technology 


The  SRL  assessment  incorporating  the  insertion  of  the  more  mature  technology  is 
shown  in  Figure  4.  By  inserting  this  more  mature  technology,  the  component  SRL  of  that 
element  has  risen  above  the  scheduled  development  point  along  with  the  component  SRLs  of 
all  of  the  technologies  with  which  it  integrates.  Previously,  these  levels  were  determined  to  be 
held  lower  although  they  were  mature  because  they  were  interfacing  with  a  lower  maturity 
component.  Additionally,  the  overall  composite  SRL  has  seen  a  dramatic  increase  as  indicated 
by  the  advancement  of  the  indicator  below  the  SRL  development  scale.  Cleary  this  example 
indicates  an  instance  in  which  the  insight  provided  by  both  component  and  composite  SRL  was 
critical  to  identifying  and  assessing  areas  of  system  development  risk. 


DEFENSE  ACQUISITION  IN  TRANSITION 


- 137- 


MP  SRL 


MP  SRL 


MP  1  0.64 


Frame 


0.67 


LEGEND 

MR  Technology 
□  Sea  Frame  System 

Current  Mission  F^ckage  SRL  Status 
Previous  Mission  Package  SRL  Status 
Current  Mission  System  SRL  Status 
Technology  Readiness  Level 
Integration  Maturity  Level 
System  Readiness  Level  Demarcation 
—  —  Scheduled  Position 

Risk  to  Cost  sndfor  Scheduie 
Low  Medium  High 


MVCS  (USV) 

USV 

BPAUV 


SRL. 


0  0 


AMNS 
ALMDS 
MH-60S  MRS 

JT7 


Figure  4.  Enhanced  Readiness  via  Capability  Trade 


Technology  Insertion/Integration  Challenge  for  Systems-of- 
systems 

There  are  many  reasons  for  the  insertion  of  new  technologies  into  existing  systems-of- 
systems.  Activities  can  range  in  scope  from  simple  obsolescence  work  focused  merely  on 
keeping  system  functionality  at  a  given  level  all  the  way  to  incremental  elements  of  continued 
development.  The  latter  case  represents  an  opportunity  for  not  only  replacement  of  an  element, 
but  also  potential  changes  to  the  existing  architecture  that  is  currently  completely  functional.  The 
impacts  of  such  insertion  can  be  equally  varied.  Ideally,  the  new  technology  or  set  of 
technologies  fit  seamlessly  with  that  of  the  old  while  increasing  performance.  However,  this  is 
seldom  the  case  and,  in  some  cases,  the  addition  can  cause  a  reduction  in  the  current  state  of 
the  system.  Other  impacts  include  forcing  its  functionality  and  performance  to  vary  widely  with 
impacts  ranging  from  cost  to  reliability,  maintainability,  and  availability.  In  most  cases,  failure  of 
technology  insertion  can  be  traced  back  to  a  common  failure  cause — integration.  This  can 
include  not  giving  proper  consideration  to  the  original  requirements  of  the  functional  system  or 
how  the  requirements  have  evolved  due  to  the  realities  of  operational  use.  Consideration  must 
be  given  early  and  often  to  how  original  systems  had  been  designed  and  what  modifications 
must  be  made  to  allow  the  integration  to  proceed.  It  is  also  important  to  note  that  two  mature 
(TRL  9),  even  operational  technologies,  will  no  longer  be  at  an  equivalent  level  of  maturity  when 
combined.  Simply  put,  significant  risk  exists  when  two  mature  products  are  brought  together  as 
the  combination  often  does  not  result  in  a  product  of  equivalent  maturity. 

Instances  in  which  integration  of  mature,  proven  technologies  can  produce  unintended 
consequences  are  numerous  both  inside  and  outside  of  DoD  acquisition.  A  perfect  example  of 
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this  occurred  with  the  Massachusetts  Bay  Transit  Authority  (MBTA)  in  the  mid  1990’s  and  took  a 
full  eight  years  to  resolve  (Fraser,  Leary  &  Pellegrini,  2003).  MBTA  operates  the  oldest  light  rail 
system  in  North  America  with  sections  dating  back  over  100  years.  In  order  to  enhance 
handicapped  accessibility  it  was  determined  that  a  new  series  of  railcars  would  be  needed.  A 
competitive  bid  was  sent  out  and  the  winner  leveraged  completely  mature  and  well  understood 
component  technologies  integrated  into  a  new  design.  A  prototype  was  constructed  and  entered 
into  testing  in  1998,  less  than  three  years  after  contract  award.  Testing  proceeded  as  planned 
and  the  design  entered  revenue  service  in  early  1999.  However,  this  entry  into  service  marked 
the  beginning  of  a  four-year  period  in  which  braking  performance  issues  and  derailments 
caused  repeated  withdrawal  from  service.  During  this  time  an  extensive  investigation  into  the 
performance  issues  was  conducted. 

The  investigation  noted  many  areas  where  integration  of  the  well-proven  technologies 
with  each  other  and  into  the  existing  system  infrastructure  introduced  unintended  issues.  These 
included  difficulties  in  matching  dynamic  car  acceleration  and  braking  performance  to  those  cars 
already  in  the  fleet  as  well  as  the  integration  of  the  new  wheel  design  to  existing  rails.  In  both 
cases  the  new  car  design  met  requirements,  but  failure  to  properly  identify  and  account  for  the 
complexities  involved  with  the  integration  of  technologies,  even  well  understood  technologies, 
caused  significant  issues.  In  this  case,  the  application  of  SRL  could  have  been  a  significant  aid 
as  it  would  have  allowed  for  the  tracking  of  the  technologies  in  the  new  design  and  their 
integration  with  one  another  as  well  as  the  overall  integration  to  the  existing  system  and 
operational  environment.  It  cannot  be  sufficiently  emphasized  that  performance  of  technology  in 
a  stand-alone  environment  does  not  mean  that  the  technology  can  be  inserted  at  a  system  level 
without  significant  planning,  monitoring,  and  assessment. 

Impact  of  Degree  of  "Design  for  Integratability"  Inherent  in  Individual  Systems 

(i.e.,  standards-based,  common  elements,  non-planned,  etc.) 

In  the  command-and-control  world,  an  approach  to  mitigate  unplanned  integration  has 
been  developed  and  is  commonly  referred  to  as  Service-oriented  Architecture.  In  this  manner,  a 
common  set  of  standards  are  used  to  define  interfaces  and  data  types  allowing  a  variety  of 
elements  from  different  developers  to  be  quickly  integrated  into  a  common  whole. 

Depending  on  the  system,  the  degree  to  which  standards  are  applied  and  designed 
inherently  brings  about  different  levels  of  integratability.  Two  development  projects  beginning 
simultaneously  can  have  drastically  different  trajectories  based  on  the  degree  to  which  the 
technologies  were  designed  to  integrate.  This  difference  can  be  seen  by  either  enhanced  IRL 
scoring  or  a  far  more  rapid  rise  through  the  TRL  maturation  process  due  to  significant  amount  of 
“pre-work”  done  via  standards  incorporation. 

Another  important  consideration  when  it  comes  to  integration  is  its  multiple  aspects. 
Depending  on  someone’s  background,  talk  of  integrating  an  avionics  box  into  an  airframe  can 
have  drastically  different  meanings.  To  a  software  engineer,  integration  means  getting  the  box 
to  exchange  data  with  the  countless  other  computers,  sensors  and  control  mechanisms  on  the 
pan.  To  an  aerospace  engineer,  integration  means  the  accounting  of  the  systems  weight  in  the 
overall  performance  of  the  plane.  To  a  mechanical  engineer,  integration  means  ensuring  the 
box  fits  in  the  rack;  to  an  electrical  engineer,  integration  means  the  type  and  amount  of  power 
required.  Even  to  a  human-factors  expert,  integration  will  mean  balancing  functionality  with  the 
pre-existing  cockpit  workflow.  While  these  examples  are  relatively  simplistic,  it  very  rapidly 
becomes  clear  that  integration  is  not  just  a  single  attribute  that  can  be  tracked  as  such;  instead, 


DEFENSE  ACQUISITION  IN  TRANSITION 


- 139- 


it  must  be  tracked  at  countless  levels  and,  indeed,  even  the  influence  of  the  different  types  of 
integration  must  be  taken  into  consideration. 

In  a  real-life  example  of  the  above  situation,  the  Army’s  canceled  Aerial  Common  Sensor 
(ACS)  program  can  be  examined.  In  this  instance,  a  highly  capable  intelligence  gathering 
system  was  to  be  integrated  onto  an  existing  airframe  design.  Early  focus  on  the  intelligence 
system  design  and  architecture  produced  a  cutting-edge  solution  that  met  or  exceeded 
customer  requirements.  However,  it  quickly  became  apparent  that  the  design  would  be  too 
heavy  for  the  selected  airframe,  and  the  program  was  subsequently  canceled  after  other 
mitigation  attempts  failed.  In  this  case,  it  was  not  the  integration  of  emerging  technologies  that 
posed  a  problem  but  rather  the  simple  matter  of  vehicle  payload,  further  underscoring  the  need 
for  a  comprehensive  architecture  analysis  and  integration  monitoring  methodology  at  all  levels 
of  systems  design. 

Consideration  of  Integration  Types 

In  cases  such  as  these,  it  is  important  to  note  that  a  single  view  of  the  previously 
discussed  network  diagram  and  SRL  assessment  may  not  be  enough.  In  the  PMS  420  program, 
IRL  criteria  have  been  broken  into  different  types  to  account  for  software  and  hardware,  in 
addition  to  physical  aspects  such  as  weight  and  clearances. 

In  essence,  it  looks  at  internal  and  external  integration  to  a  SoS.  These  can  be 
considered  individually  for  greater  detail  or  summed  up  in  a  roll-up  chart  to  appease 
management.  In  this  way,  countless  implications  and  variations  of  integration  can  be  tracked  in 
a  single  place. 

Reduction  of  Integration  Risk  in  a  System-of-systems 

As  discussed  above,  integration  of  components  is  one  of  the  key  areas  of  risk  for 
developmental  and  production  activities.  While  the  SRL  methodology  will  provide  insight  into  the 
potential  risk  for  mangers  to  understand,  it  does  not  inherently  provide  methods  for  reducing  the 
risk.  One  way  PMS  420  is  seeking  to  reduce  that  risk  is  through  the  increased  use  of  common 
components  across  the  SoS  in  order  to  drive  down  integration  uncertainty.  Basically,  an 
expansion  of  the  open  architecture  concept,  PMS  420  is  seeking  to  define  and  manage  the 
interfaces  to  be  used  by  concepts  seeking  evaluation  for  insertion  while  allowing  the  technical 
capability  to  mature/change  internally  to  the  externally  defined  interfaces.  An  example  of  this 
process  is  how  all  mission  package  services  were  devised  to  operate  upon  a  common  operating 
system  on  common  hardware.  Within  an  individual  mission  package  this  capability  was  further 
allocated  to  individual  mission  systems,  thereby  providing  processing  and  storage  capabilities 
while  requiring  they  support  a  minimal  level  of  integration  for  use  of  common  core  capabilities. 
The  drive  towards  Service  Oriented  Architectures  for  software  base  capabilities  is  another 
example  of  how  integration  risk  can  be  minimized  by  increasing  the  use  of  common  capabilities 
vice  having  each  system  try  to  provide  an  end-to-end  solution. 

Future  Planned  Expansion  of  the  SRL  Methodology 

While  the  use  of  the  present  SRL  methodology  described  above  has  helped  the  Mission 
Modules  Program  in  terms  of  effectively  managing  system-of-systems  procurement  by  providing 
additional  insight  into  the  technical  and  integration  risks  associated  with  the  incremental 
acquisition  or  spiral  development  of  capabilities,  the  efforts  to  date  are  just  beginning  to  scratch 
the  surface  of  providing  management  with  the  information  required  to  make  informed  decisions 
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and  to  apply  these  decisions  in  a  predictive  method  for  selecting  technologies  for  future 
increments  and  spirals.  Several  areas  have  been  identified  as  areas  of  investigation  designed  to 
further  increase  management  insight  in  helping  to  resolve  these  deficiencies.  These  focus  areas 
include  the  incorporation  of  methodologies  designed  to  allow  for  the  program  office  to  gain 
better  insight  on  the  impact  of  inserting  a  new  technology  across  the  spectrum  of  the  SoS’s 
existing  performance  capabilities,  the  inclusion  of  cost  factors  and  monitoring  into  the  tool  to 
allow  for  both  predictive  determination  of  “should  cost”  factors,  and  the  use  of  the  tool  to  provide 
insight  into  cost  versus  performance  status  monitoring.  Additionally,  for  the  Mission  Module 
Program  there  is  a  desire  to  increase  the  use  of  common  components  across  the  warfare  areas. 
This  drive  for  commonality  could  impact  performance  and  a  method  of  analyzing  the  cost 
benefits  versus  performance  risks  prior  to  implementation  is  needed.  All  these  focus  areas  are 
areas  of  growth  for  the  methodology  and  will  be  discussed  in  the  following  paragraphs. 

Technology  Insertion/Integration  Focus  Area 

One  of  the  challenges  of  managing  technology  insertion  into  spiral  or  incremental 
programs  is  determining  the  value  added  and  understanding  the  potential  of  a  capability  lost  by 
inserting  a  new  capability.  Historically,  technology  insertion  into  a  stand-alone  system  has  only 
focused  on  the  cost  versus  capability  gained  determination.  In  a  system-of-systems,  especially 
a  constrained  system-of-systems  design  such  as  the  mission  packages,  the  value  of  the 
capability  gained  on  a  individual  system  has  to  be  assessed  in  terms  of  the  impact  and  cost  to 
that  system  as  well  as  to  the  entire  system  of  system.  For  example,  Figure  4  is  an  example  of 
how  technology  blocks  for  the  MVCS  control  the  present  limitation  on  how  far  an  unmanned 
surface  vehicle  (USV)  can  be  from  the  LCS.  A  new  manufacturer  may  devise  a  new 
communications  capability  that  can  greatly  enhance  the  USV’s  operational  range  without 
increasing  its  cost  or  weight.  While  initially  a  great  potential  for  improvement,  the  effectiveness 
of  implementing  the  change  is  only  beneficial  if  the  greater  range  can  be  utilized  and  the  impact 
of  incorporating  it  does  not  impact  the  ability  of  the  package  to  conduct  all  of  its  assigned 
missions.  Thus,  the  impacts  and  limitations  imposed  by  the  directly  linked  components  of  the 
USV  need  to  be  understood  but,  more  importantly,  the  total  Mission  Package  impacts  need  to 
be  understood.  If  the  new  capability  added  sufficient  weight  to  the  USV  string,  shown  in  Figure 
4,  it  might  create  a  condition  in  which  the  total  weight  of  the  package  exceeded  limitations,  and 
a  sensor  might  have  to  be  removed  from  the  helicopter  to  remain  within  the  weight  constraint. 
The  loss  of  the  sensor  might  mean  that  the  Mission  Package  could  no  longer  complete  all 
assigned  missions — so  what  appeared  to  be  an  improved  capability  at  the  start  can  turn  into  a 
negative  if  the  impacts  are  not  understood  early  enough  to  enable  informed  decision-making. 
One  method  that  PMS420  is  investigating  for  asserting  this  determination  is  using  reliability 
block  diagrams  developed  by  the  mission  packages  to  predict  end-to-end  mission  capability 
reliability,  shown  in  Figure  5,  and  overlaying  the  TRL’s  and  IRL’s  development  through  the  SRL 
methodology  to  increase  the  understanding  of  the  risk  areas  involved  across  a  package  when 
deciding  to  implement  changes. 
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Figure  5.  PMS  420  Reliability  Block  Diagram 


Cost  Prediction  and  Monitoring 

Up  until  now  we  have  been  talking  about  the  engineering  implications  of  integration.  As 
well  all  know,  however,  the  real  world  is  about  more  than  just  technical  development.  In  order  to 
have  a  successful  system  development  effort,  it  is  imperative  that  the  design  not  only  function, 
but  also  be  built  at  the  price  that  is  affordable  to  the  buyer.  This  is  especially  critical  at  a  time 
when  acquisition  costs  have  soared,  and  it  seems  that  even  the  most  well-understood  jobs 
cannot  be  completed  on  time  or  within  budget.  A  fundamental  failure  in  this  area  again  relates  to 
integration.  Though  the  unique  art  and  science  that  is  cost  estimation  has  been  steadily 
expanding  in  experience  for  decades,  the  knowledge  available  for  appropriately  modeling  and 
estimating  the  level  of  effort  required  to  integrate  various  pieces  of  technology  into  a  holistic 
capability  is  limited. 

Degree  of  "Design  for  Integratability"  Dramatically  Impacts  Cost  Estimates 

As  with  the  previous  examples  of  integration  maturity  being  at  different  levels  based  on 
the  degree  to  which  technologies  have  been  designed  to  integrate,  cost  is  a  similar  and,  in 
some  sense,  even  more  complicated  operation.  While  there  is  a  significant  amount  of  data 
available  to  determine  the  cost  of  a  new  ship  based  on  its  displacement  or  on  a  new  piece  of 
software  based  on  its  lines  of  code,  the  understanding  of  how  these  two  elements  connect  is  far 
less  understood.  From  the  perspective  of  a  cost  estimator,  who  in  many  cases  is  outside  the 
bounds  of  the  program,  the  degree  of  work  it  takes  to  modify  a  pair  of  technologies  to  work 
together  is  somewhat  of  a  mystery.  The  systems  that  have  been  designed  with  a  standards- 
based  approach  may  require  little  more  than  being  brought  together  whereas  other  systems 
may  require  significant  modification  and  extension  of  existing  documents. 
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Cost  Estimation  Relationship  Plans 


In  order  to  better  capture  this  and  estimate  it  for  the  PMS  420  effort,  steps  are  being 
taken  to  categorize  the  integration  at  hand.  In  this  way,  a  standards-based  approach  can  be 
costed  based  on  similar  historical  efforts  while  a  dramatic  revamping  can  be  costed  entirely 
differently.  This  information  is  being  combined  with  technical  status  data  to  form  an  overarching 
assessment  tool  known  as  a  System  Maturity  Model.  In  this  effort,  inputs  on  both  technical 
development  and  factors  impacting  cost  are  collected  side-by-side  with  cost  and  technical 
development  information.  Care  is  taken  to  specify  the  data  types  requested  and  examples  are 
provided  to  ensure  that  the  responses  received  are  the  type  and  quality  requested.  This 
requirement  for  data  is  then  forwarded  to  subject  matter  experts  for  input.  The  data  is  then 
collected  and  used  to  populate  algorithms  that  produce  cost  and  technical  assessments  for  the 
program.  In  the  near-term,  these  assessments  take  the  form  of  the  CARD,  PLCCE,  and 
Milestone  B  documentation,  but  over  the  life  of  the  program  they  will  also  form  the  foundation  of 
program  status  reports  and  monitoring  tools.  By  combining  this  information  together  from  a 
system-of-systems  perspective,  the  interdependencies  of  cost  and  technical  development  from 
a  holistic  perspective  can  be  most  accurately  captured.  As  outlined  above,  the  technical  and 
integration  maturity  of  the  components  are  used  to  form  the  basis  of  cost  estimates  for 
development.  From  there,  a  variety  of  other  information  can  be  applied  to  expand  that  initial 
acquisition  cost  model  into  an  estimate  of  total  program  lifecycle  cost.  Key  elements  include  the 
operations  and  maintenance,  CONOPS,  as  well  as  the  technology  insertion  and  obsolescence 
pans.  By  leveraging  the  architecture  diagrams  of  how  future  technologies  will  be  applied  to  the 
system  over  the  course  of  time,  a  more  accurate  assessment  can  be  obtained.  A  significant 
amount  of  unknowns  and  guess  work  can  be  reduced  since  detailed  plans  for  what  technologies 
may  be  inserted  and  in  what  manner  that  may  happen.  This  eliminates  surprise  modernization 
service  life  extension  efforts  later  in  the  program  from  running  wildly  out  of  control. 

Conclusion 

System-of-systems  development  is  here  to  stay  and  will  undoubtedly  only  grow  more 
complex  as  the  technologies  that  make  up  the  systems  continue  to  evolve,  expand,  and  push 
the  leading  and  often  bleeding  edge  of  technology.  With  this  evolution,  complex  systems 
integration  begins  to  require  a  paradigm  shift  in  how  assessment,  analysis,  and  management 
techniques  must  be  used  and  what  tools  are  applicable.  No  longer  can  the  development  of 
individual  technologies  be  considered  in  isolation;  rather,  these  developments  and  their 
integration  with  one  another  must  be  defined  and  analyzed  in  new  and  enhanced  ways.  It  is  only 
by  considering  the  impacts  of  all  technologies  and  integrations  as  a  whole  that  the  acquisition 
approach  can  be  improved.  As  discussed  above,  the  implementation  of  a  methodology  that 
combines  assessment  of  the  technology  maturity  of  component  pieces  with  the  assessment  of 
their  integration  level  has  been  shown  to  add  value.  The  next  step  in  improving  this  technique  is 
to  continue  to  expand  its  use  in  assisting  in  technology  insertion  assessments  by  using  it  as  a 
predictive  tool.  Beyond  that,  the  goal  is  to  incorporate  cost  inputs  into  the  tool  to  provide  further 
insight  to  management  on  the  existing  risks  thereby  being  accepted  in  the  selection  of 
technologies  for  incorporation  into  mature  systems-of-systems. 
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Overview 


•  Unique  System  of  Systems  (SoS)  Acquisition  Management  Needs 

•  LCS  Mission  Package  Development  -  a  true  SoS 

•  System  Readiness  Level  (SRL)  Development  /  Implementation 

•  Applications  in  Management  Decision  Making 

•  Technology  Insertion  in  SoS’s 

•  Case  Study  -  Considerations  for  Legacy  Systems 

•  Future  Developments  -  Risk  Monitoring 

•  Future  Developments  -  Cost  Profiles 

•  Conclusion  /  Lessons  Learned 
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Unique  SoS  Acquisition  Management  Needs 

•  SoS  acquisition  management  represents  a  significant  increase  in 
complexity  over  traditional  system  acquisition 

•  Development  requires  that  significant  numbers  of  new  and  existing 
technologies  be  integrated  to  one  another  in  a  variety  of  ways 

•  Poses  challenges  to  traditional  development  monitoring  tools  and 
cost  models  due  to  the  need  to  capture  integration  complexity  and 
the  level  of  effort  required  to  connect  individual  components 

•  A  high  degree  of  inter-linkage  between  components  can  also  cause 
unintended  consequences  to  overall  system  performance  as 
components  are  modified  and  replaced  throughout  the  system  life 
cycle 

The  result  of  this  acquisition  management  paradigm 

shift  has  been  significant  schedule  and  cost 
overruns  in  SoS  programs 


LCS  Mission  Packages...  truly  a  SoS  undertaking 


Surface  Warfare  Mine  Countermeasures  Anti-Submarine  Warfare 
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Defining  Program  Office  Role  and  Needs 


•  PEO  LMW  /  PMS  420  is  responsible  for  the  development  and 
integration  of  a  series  of  Mission  Modules  to  be  used  on  the  Littoral 
Combat  Ship 


•  Modules  leverage  considerable  amounts  of  technology  from  existing 
programs  of  record  while  also  conducting  new  development 


•  Keys  aspects  of  the  project  include  not  only  monitoring  the  status  of 
technology  development,  but  also  the  maturity  of  the  numerous 
integrations  between  those  technologies  and  external  interfaces 


•  This  has  resulted  in  a  very  complex  and  diverse  system  of  systems 
engineering  activity  with  a  need  to  obtain  quick  and  accurate 
snapshots  of  development  maturity  status,  risks,  and  issues 


TRL  Shortcomings 


•  Application  of  TRL  to  systems  of  technologies  is  not  sufficient  to  give 
a  holistic  picture  of  complex  SoS  readiness 

-  TRL  is  only  a  measure  of  an  individual  technology 

•  Assessments  of  several  technologies  rapidly  becomes  very  complex 
without  a  systematic  method  of  comparison 


•  Multiple  TRLs  do  not  provide  insight  into  integrations  between 
technologies  nor  the  maturity  of  the  resulting  system 

-  Yet  most  complex  systems  fail  at  the  integration  points 


Individual  Technology 


System  of  Technologies 
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Methodology  Development  Overview 


GOAL:  Institute  a  robust,  repeatable,  and  agile  method  to  monitor  /  report 
system  development  and  integration  status 


Create  a  System  Readiness  Level  (SRL)  that  utilizes  SME  /  developer 
input  on  technology  and  integration  maturity  to  provide  an  objective 
indication  of  complex  system  development  maturity 


Status  of  connections 
between  the  technologies 


System  Readiness 
Levels  (SRL) 


Overall  system  maturity 
appraisal 


•  Provides  a  system-level  view  of  development  maturity  with  opportunities  to  drill  down 
to  element-level  contributions 


•  Allows  managers  to  evaluate  system  development  in  real-time  and  take  proactive 
measures 

•  Highly  adaptive  to  use  on  a  wide  array  of  system  engineering  development  efforts 

•  Can  be  applied  as  a  predictive  tool  for  technology  insertion  trade  studies  and  analysis 
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SRL  Methodology  Decomposition  for  PMS  420 


Step  1 :  Identify  hardware  and 
software  components 


Step  2:  Define  network  diagram 
for  systems 


Step  3:  Define  system 
operational  threads  (If  applicable) 


► 
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Technology  1  1 
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Technology  2  1 

t  , 
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Technology  4  1 

1 

Technology  5  | 

► 


Include  all  technologies  that  make-up 
the  overall  system 


Emphasis  is  on  the  proper  depiction  of 
hardware  and  software  integration 
between  the  components 


Thread  analysis  allows  for  the  option  of 
weighting  the  most  important 
components  and  evaluation  of  alternate 
operational  states 


Step  4  Apply  detailed  TRL  and  IRL 
evaluation  criteria  to  components 
and  integrations 


Initial  Architecture  Definition  and  Setup 

Step  5:  Calculate  individual  and  Step  6:  Document  status  via  roll- 


composite  SRLs 


up  charts 


Integration  Maturity  Level  1 

ioo  Components  to  be  integrated  are  selected 
ioo  Component  interface  points  are  identified 
ioo  Documented  phase/build  plan  for  component  availability 

inn  Rata  tvnftc  irlentifiorl 


%  [Technology  Readiness  Level  1~ 


ioo _ Physical  laws  and  assumptions  used  in  new  technologies  defined 

Have  some  concept  in  mind  that  may  be  realizable  in  software 
Know  what  software  needs  to  do  in  general  terms 
Paper  studies  confirm  basic  principles  and  system  concepts 

_ Mathematical  formulations  of  concepts  that  might  be  realizable  in  software 

ioo  Have  an  idea  that  captures  the  basic  principles  of  a  possible  algorithm 
Basic  scientific  principles  observed 

ioo _ Research  hypothesis  formulated 

Identify  who  will  perform  research  and  where  it  will  be  done 
60  Readiness  Level  Percent  Complete  (non-weighted) 


Checklist  style  evaluation  allows  for  the 
ability  to  “take-credit”  for  steps  that  have 
taken  place  beyond  the  current  readiness 
level 


NOtT!  ALL  CUTJl  H  THEE  TBURjirfIS  MOTKjHAL 

System  Detailed  Status 


Input  TRL  and  IRL  evaluations  into 
algorithm  to  compute  an 
assessment  of  overall  system 
status  via  SRLs 


Populate  reporting  chart  templates 
with  evaluation  and  calculation 
outcomes  to  highlight  both  current 
status  and  performance  over  time 


Iterative  SME  Evaluation  Throughout  Development  Cycle 


SRL  Reporting  Method  for  PMS  420 


Concept  Feasibility  Basic  Technology  System  System  Demo  System  to  Qualification  DT/OT  Operational 

Definition  Demonstration  Technology  Testing  Integration  and  Test  System  Testing  Complete  System  Mission 


Integration  Integration  Proven 


•  For  complex  systems,  the  amount  of  information  obtained  from  the  SRL 
evaluation  can  be  overwhelming 

•  To  maximize  applicability  SRL  outputs  are  tied  to  key,  program-specific 
development  milestones 

•  Progress  against  these  milestones  provide  key  insight  to  the  user  regarding 
current  program  development  maturity  status,  risk,  and  progress 
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Applications  in  Management  Decision  Making 

•  Current  development  status  monitoring 

-  Enables  monitoring  of  system  technology  maturation  with  all  integrations 
considered 

-  Enables  a  prioritization  of  technology  development  maturity  for  each 
component  of  the  system 


•  Decision  making 

-  Allows  components  identified  as  “lagging”  to  be  analyzed  further  for  root 
cause 

-  Resources  can  be  more  properly  distributed  to  those  technologies  in  need 

-  Impacts  can  be  examined  by  quickly  analyzing  multiple  “what-if  scenarios 

-  Allows  projected  maturity  changes  to  be  examined  along  with  cost  and 
schedule 

in  complex  SoS  efforts  it  is  not  always  immediately  clear 
where  resources  should  be  applied  for  maximum  fcj,  j 
gains  in  maturity  and  reductions  in  risk 
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Effectively  Channeling  Resources 
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6  months  later. . . 

A  re-distribution  of  resources  has 
allowed  the  lagging  component  to 
catch-up 
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Technology  Insertion  in  SoS’s 

As  with  the  monitoring  of  current  status  in  SoSfsf  the 
process  of  technology  identification 9  analysis  and 

insertion  is  also  made  considerably  more  complex 

Key  Questions  to  Consider  Include: 

-  Which  of  the  existing  components  of  the  system  should  be  either 
replaced  or  enhanced? 

-  How  will  the  new  technology  be  integrated  into  the  system? 

-  What  are  the  types  of  integration  involved? 

•  Logical  /  Data  flow 

•  Physical 

•  Functional 

•  Human-to-Machine 

-  What  is  the  projected  impact  on  performance?  (How  do  we  optimize?) 

-  Are  there  any  legacy  design  constraints  that  will  impact  selection? 


Case  Study  -  Considerations  for  Legacy  Systems 


•  Background: 

-  Massachusetts  Bay  Transit  Authority  needed  new  light  rail  cars  to 
enhance  handicapped  access 


•  Legacy  System  Description: 

-  Oldest  light  rail  system  in  North  America  with  some  infrastructure  dating 
back  over  100  years 

-  New  cars  would  need  to  operate  in  conjunction  with  existing  rolling  stock 


•  Design  Solution: 

-  Leveraged  completely  mature  and  well 
understood  component  technologies  in  a 
new  design 

•  Outcome: 

-  Fielded  prototype  experienced  four  years 
of  braking  performance  issues  and 
derailments  causing  repeated  withdrawals 
from  service 


SOURCE:  Fraser,  G.R.,  Leary,  R.J.,  Pellegrini,  M.M.C.,  Integrating  New  Light  Rail  Vehicle  Technology  in  Mature 
Infrastructure,  Transportation  Research  Circular  EC-058,  9th  National  Light  Rail  Transit  Conference. 
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Case  Study  -  What  Went  Wrong??? 


•  Well  proven  technologies  integrated  with  one  another  in  new  ways 
and  into  an  existing  infrastructure  created  unintended  issues 
including: 

-  Difficulties  in  matching  the  new  car’s  acceleration  and  braking 
performance  to  existing  car’s  capabilities  due  to  inherent  characteristics 
of  technologies  employed 

-  Introduction  of  an  “advanced”  wheel  design  that  was  unable  to 
accommodate  an  infrastructure  that  has  deviated  from  original  design 
specifications  over  years  of  use 

•  In  all  cases  the  design  met  requirements,  but  failed  to  adequately 
accommodate  the  constraints  imposed  by  the  overall  system  and 
environment 

Performance  of  a  technology  in  a  stand-alone  environment  does 
not  mean  that  the  technology  can  be  inserted  at  the  system  level 
without  significant  planning,  monitoring,  and  assessment 


Future  Developments  -  Understanding  Tech 
Insertion  Impact 


Insertion  considerations  for  new 
components  must  be  based  not  only 
on  the  projected  impact  on  a  given 
capability,  but  on  all  of  the 
capabilities/missions  of  the  SoS 

-  In  some  instances  it  is  conceivable 
that  the  negative  impact  on  the  overall 
system  outweighs  the  gains  in  a 
single  area  of  operation 

Various  options  exist  for  laying  out 
SoS  Mission  Definitions 

-  One  option  is  using  existing  end-to-end 
reliability  block  diagrams  developed  for 
RMA  analysis  with  SRL  assessment  inputs 
to  increase  overall  understanding  of 
decisional  impacts  across  the  system 


Example-change  of  USV  design 
impacts  3  mission  areas  and  3 
interfacing  sensors.  Are  all 
impacts  understood? 
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Trading  Off  Technology  Insertion  Options 


MVCS 

(RMMV) 


MVCS  (USV); 
BPAUV  PC 


SRL, 


MVCS  (OB) 


MH-60S; 

US3;  MH-60S  MPS 
BPAUV 


MP  SRL 

MP  SRL 

w/o  Sea  Frame 

MP  1 

0.60 

0.57 

LEGEND 

|  1  MP  Technology 

I  |  Sea  Frame  System 

Current  Mission  Package  SRL  Status 
Previous  Mission  Package  SRL  Status 
Current  Mission  System  SRL  Status 
Technology  Readiness  Level 
Integration  Maturity  Level 
System  Readiness  Level  Demarcation 
—  —  Scheduled  Position 

Risk  to  Cost  and/or  Schedule 
Low  Medium  High 


AQS-20 


AMNS; 

ALMDS 


^7  *V 
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Taking  Action  to  Mitigate 


System  maturity  is  enhance df  advanced 
capability  will  be  employed  in  future  evolutions 


MVCS  (OB) 

MVCS  (USV) 

DLS  (OB) 

USV 

BPAUV 

BPAUVPC  DLSjjRMMV) 


€> 
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Future  Developments  -  Cost  Profiles 


•  PEO  LMW  /  PMS  420  is  working  with  NAVSEA  05C  (NAVSEA’s 
cost  analysis  division)  to  develop  a  life  cycle  cost  model  specifically 
tailored  to  SoS  analysis 

•  Factors  contributing  to  costs  in  SoS 

-  Integration  type  (physical,  functional,  logical) 

-  Use  of  standards  (Were  components  designed  to  integrate?) 

-  Maturity  of  technologies  being  integrated 

•  A  correlation  between  the  SRL  and  cost  numbers  may  bring  about 
the  ability  to  track  actual  development  maturity  vs.  costs 

•  Linkage  to  technology  trade-off  and  planning  environments  allows 
cost  to  be  analyzed  in  consideration  with  maturity  and  performance 
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Initial  SRL  Implementation  Lessons  Learned 


•  Methodology  is  highly  adaptable  and  can  be  quickly  applied  to  a 
wide  variety  of  development  efforts 

•  Programs  tend  to  minimize  the  importance  of  system  and  subsystem 
integration  and  thus  overestimate  the  maturity  of  their  development 

•  Widespread  familiarity  with  TRL  makes  acceptance  and  utilization  of 
TRL  and  IRL  easier 


•  Formulating  the  system  architecture  early  in  development  is  a  key 
step  and  leads  to  an  enhancement  of  the  overall  systems 
engineering  effort 


•  System  architecture  formulation  also  provides  the  opportunity  to 
bring  together  SMEs  from  both  the  physical  and  logical  realms  and 
necessitates  insightful  discussions  across  the  team 


•  The  decision  maker  is  afforded  the  ability  to  assess  program  status 
from  a  system  of  systems  perspective 


The  SRL  methodology  delivers  a  holistic  evaluation  of  complex 
system  readiness  that  is  robust,  repeatable,  and  agile 
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Conclusions 


•  SoS  development  represents  a  new  level  of  challenge  in 
acquisition  management 

•  SRL  provides  one  possible  assessment,  analysis  and 
management  technique 

•  Methodology  leads  to  holistic  monitoring  of  all  factors 
impacting  system  development 

•  Future  work  includes  extending  the  concepts  for 
understanding  cost  impacts  (CAIV)  in  an  incremental 
acquisition 
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“String”  Analysis  Incorporated 


Complex  systems  often  offer  numerous  options  for  conducting  operations 


•  Operational  strings  were  created  that  identified  the  components 
required  to  utilize  a  single  function  of  the  system 

•  Assessment  of  the  SRL  for  each  of  these  options  allows  for  a  better 
understanding  of  the  maturity  of  each  operating  configuration 


•  Understanding  the  true  status  of  the  system  on  an  operational  string 
level  allows  for  the  opportunity  to  field  initial  capability  earlier  and 
then  add  to  it  as  other  strings  mature 
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Basic  SRL  Calculators  Developed 


•  Calculators  are  developed  and  defined  for  the  system  being  evaluated 

•  Allows  for  real-time  updates  to  TRL  and  IRL  inputs  and  the  resulting  SRL 
evaluation  providing  decision-makers  with  instant  feedback  on  “what  if 
scenarios 

•  Intuitive  interface  removes  the  need  for  the  user  to  manipulate  and  deal 
with  the  mathematics  of  the  SRL  calculation 


Architecture  1  System  Readiness  Level  Calculator 

S yst e 111  S 1 1  i  11  (J  •*M8B 

TRL  SRL  SRL  Lf 


Notional  Interface  Diagram 


Technology  1 
Technology  2 
Technology  3 
Technology  4 
Technology  5 
Technology  5 
Technology  7 
Technology  8 
Technology  9 
Technology  10 
Technology  11 A 

Hub 

Technology  12 
Technology  13 
Technology  14 
Technology  15 
Technology  16 
Technology  17 
Technology  18 
Technology  19 
Technology  20 


Composite  SRL  (1,9) 

6.8 

Composite  SRL  (0,1)  Q 

3  0.76 

System  SRL  -  Reflective  of  all  of  the  technologies  and  integrations  of  the  system  considered  at  once 

String  SRL  -  Reflective  of  only  those  technolgoues  and  integrations  used  in  an  operational  string 
(Strings  are  named  by  the  final  technology  in  the  string,  but  the  number  shown  is  actually  the  composite  SRL  for  that  string) 
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Verification  and  Validation  Activities 


IRL  Criteria 


•  Created  expanded  list  of  IRL 
criteria  for  each  readiness 
level 

•  Goal  was  to  capture  the  key 
elements  of  the  integration 
maturation  process 

•  Presented  to  30  integration 
SMEs  from  across 
government,  academia,  and 
industry 

•  Asked  to  assess  importance  of 
each  criterion 

•  Results  show  solid  buy-in 
among  SMEs  that  identified 
criteria  are  key  factors  in 
successful  integration 


SRL  Evaluation  Process 

•  Conducted  a  “blind  trial”  of 
SRL  methodology  and 
evaluation  process 

•  User’s  Guide  and  evaluation 
criteria  were  sent  to  key 
system  SMEs 

•  From  just  these  resources 
SMEs  were  asked  to  conduct 
the  evaluation  and  report  on 
the  results 

•  Compiled  results  and  iterated 
on  lessons  learned  to  improve 
the  process 


efense  Acquisition  in  Transition 

£?  ANNUAL  ACQUISITION  RESEARCH  SYMPOSIUM 


May  12-14,  2009 
Monterey,  CA 


SRL  Calculation 


•  The  SRL  is  not  user  defined,  but  is  instead  based  on  the  outcomes 
of  the  documented  TRL  and  IRL  evaluations 


•  Through  mathematically  combining  these  two  separate  readiness 
levels,  a  better  picture  of  overall  complex  system  readiness  is 
obtained  by  examining  all  technologies  in  concert  with  all  of  their 
required  integrations 

SRL  =  I RL  x  TRL 


r 

r 

IRL11 

IRLi2 

"\ 

IRL13 

r  ~\ 

TRLj 

SRL, 

V. 

SRL2 

srl3 

J 

IRL12 

JRL13 

irl22 

IRL23 

IRL23 

irl33^ 

X 

TRL, 

TRLg 

C  J 

Composite  SRL  =  1/n  SRL,/ n  +  SRL2/11  +  SRL,/ n 
=  1/n2 1  SRL,  +  SRL,  +  SRL,] 

•  These  values  serve  as  a  decision-making  tool  as  they  provide  a 
prioritization  guide  of  the  system’s  technologies  and  integrations 
and  point  out  deficiencies  in  the  maturation  process 
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SRL  Calculation  Example 


TRL,  = 


9 


IRL-1,2  =  1 


TRU=  6 


TRL  Matrix 


IRL2,3  =  7 


TRL,  =  6 


I RL  Matrix 


r 

TRLX 

r 

9 

IRLj 

irl12 

1  rl13 

r9 

1 

0^ 

trl2 

— 

6 

irl12 

irl2 

irl23 

— 

1 

9 

7 

trl3 

v.  V 

6 

V.  J 

J  RL13 

irl23 

irl3 

1° 

7 

9J 

SRL  =  I  RL  x  TRL 

(NorrmH^^l 


r 

r 

SRLX  SRL2 

SRLj 

J 

0.54 

0.43 

0.59 

J 

Component  SRL*  represents  Technology  “X"  and  its  I  RLs  considered 

Composite  SRL  =  V  3  (  0.54  +  0.43  +  0.59  )  =  0.52 

The  Composite  SRL  provides  an  overall  assessment  of  the  system  readiness 

f  ■  | 


Sauser,  B.,  J.  Ramirez-Marquez,  D.  Henry  and  D.  DiMarzio.  (2007).  “A  System  Maturity  Index  for  the  Systems 
Engineering  Life  Cycle.”  International  Journal  of  Industrial  and  Systems  Engineering.  3(6).  (forthcoming) 
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Detailed  SRL  Calculation  Example 

Matrix  Setup 


The  computation  of  the  SRL  is  a  function  of  two  matrices: 

-  The  TRL  Matrix  provides  a  blueprint  of  the  state  of  the  system  with  respect  to  the 
readiness  of  its  technologies.  That  is,  TRL  is  defined  as  a  vector  with  n  entries  for 
which  the  /th  entry  defines  the  TRL  of  the  /th  technology. 

-  The  IRL  Matrix  illustrates  how  the  different  technologies  are  integrated  with  each 
other  from  a  system  perspective.  IRL  is  defined  as  an  nxn  matrix  for  which  the 
element  IRL//  represents  the  maturity  of  integration  between  the  /  th  and  j  th 
technologies. 

Populate  these  matrices  with  the  appropriate  values  from  the  previously 
documented  TRL  and  IRL  component  evaluations  and  then  normalize  to  a  (0,1) 
scale  by  dividing  through  by  9 


For  an  integration  of  a  technology  to  itself  (e.g.  IRLnn)  a  value  of  "9"  should  be 
placed  in  the  matrix 


For  an  instance  of  no  integration  between  technologies  a  value  of  "0”  should  be 
placed  in  the  matrix 


M,.,  = 

~TRLX~ 

trl2 

~IMLU 

IMLn 

IMLn 

iml22 

...  IMLIrl  ~ 
...  IMLln 

TRLn_ 

JMLnl 

IMLnl 

...  IML,m_ 

Decision  Support  Metrics  for  Developmental  Life  Cycles,  Users  Guide:  Version  2.0,  Northrop  Grumman  Corp. 
and  Stevens  Institute  of  Technology,  5  September  2007 
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Detailed  SRL  Calculation  Example 

Calculation 

•  Obtain  an  SRL  matrix  by  finding  the  product  of  the  TRL  and  IRL 
matrices 


•  The  SRL  matrix  consists  of  one  element  for  each  of  the  constituent 
technologies  and,  from%wnta§V#tl^  quantifies  the 

readiness  level  of  a  specific  technology  with  respect  to  every  other 
technology  in  the  system  while  also  accounting  for  the  development 
state  of  each  technology  through  TRL.  Mathematically,  for  a  system 
with  n  technologies,  [SRL]  is: 


~SRL{~ 

" IMLnTRLl  +  IMLl2TRL2  + ...  +  IMLlnTRLn  ~ 

[srl]= 

srl2 

— 

IML2lTRL{  +  IML22TRL2  + ...  +  IML2nTRLn 

SRLn 

IMLJRL  +  IMLSRL,  + ...  +  IMLTRLn 

nvv  nl  I  nn  n 

Decision  Support  Metrics  for  Developmental  Life  Cycles,  Users  Guide:  Version  2.0,  Northrop  Grumman  Corp. 
and  Stevens  Institute  of  Technology,  5  September  2007 


efense  Acquisition  in  Transition 

£?  ANNUAL  ACQUISITION  RESEARCH  SYMPOSIUM 


May  12-14,  2009 
Monterey,  CA 


Detailed  SRL  Calculation  Example 

Analysis 


•  Each  of  the  SRL  values  obtained  from  the  previous  calculation 
would  fall  within  the  interval  (0,  #  of  Integrations  for  that  Row). 
For  consistency,  these  values  of  SRL  should  be  divided  by  the 
number  of  integrations  for  that  row  of  the  matrix  to  obtain  the 
normalized  value  between  (0,1).  (e.g.  if  there  are  four  non-zero 
numbers  in  the  IRL  matrix  for  that  row,  divide  by  four) 

•  This  number  should  then  be  multiplied  by  9  to  return  to  the 
familiar  (1,9)  scale 

•  For  Example: 


r 

IRL, 

1  RI-1.2 

!RL13 

0 

1 

0^ 

IRL-,2 

irl2 

irl23 

= 

1 

0 

7 

IRCb 

1  RL"23 

IRLsy 

1° 

7 

°J 

4  1  Integration  (Divide  SRL  for  that  Row  by  1  and  multiply  by  9) 

4  2  Integrations  (Divide  SRL  for  that  Row  by  2  and  multiply  by  9) 

<  1  Integration  (Divide  SRL  for  that  Row  by  1  and  multiply  by  9) 


Decision  Support  Metrics  for  Developmental  Life  Cycles,  Users  Guide:  Version  2.0,  Northrop  Grumman  Corp. 
and  Stevens  Institute  of  Technology,  5  September  2007 
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Detailed  SRL  Calculation  Example 

Analysis 


SRL  2  SRL3 


•  These  individual  values  serve  as  a  decision-making  tool  as  they 
provide  a  prioritization  guide  of  the  system’s  technologies  and 
integrations  and  point  out  deficiencies  in  the  maturation  process 

•  The  composite  SRL  for  the  complete  system  is  the  average  of 
all  normalized  SRL  values.  (Note  that  weights  can  be 
incorporated  here  if  desired.) 


( SRL.  SRL ,  SRL„ } 

- L  + - L  +  '"  + - !L 

on  T  _  V  ^ ^ ^  ) 

^  Composite 

Tl 

•  A  standard  deviation  can  also  be  calculated  to  indicate  the 
variation  in  the  system  maturity 


Decision  Support  Metrics  for  Developmental  Life  Cycles,  Users  Guide:  Version  2.0,  Northrop  Grumman  Corp. 
and  Stevens  Institute  of  Technology,  5  September  2007 
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SRL  Calculation  Example 

Normalizing  the  TRLs  and  IRLs 
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Populate  with 
Evaluation  Results 


Non- Normalized  [(1,9)  scale] 
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Remember. . .  a  technology  integrated  with  itself 
receives  an  iRL  value  of  9  (e.g.  iRLn), 
while  technologies  for  which  there  is  no  connection 
between  them  receive  a  value  ofO  (e.g.  i  RL13). 


Sauser,  B.,  J.  Ramirez-Marquez,  D.  Henry  and  D.  DiMarzio.  (2007).  “A  System  Maturity  Index  for  the  Systems 
Engineering  Life  Cycle.”  International  Journal  of  Industrial  and  Systems  Engineering.  3(6).  (forthcoming) 
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Normalized  [(0,1)  scale] 
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SRL  for  System  Alpha 

Calculating  the  SRL  and  Composite  Matrix 

SRL  =  I RL  x  TRL 


Component  SRL 

SRLX  SRL2  SRL3 

v.  J 

SRL-l  srl2  srl3 

v.  J 


1.07 

1.30 

1.19 

0.54 

0.43 

0.59 

(0,n)  scale 


■J  Where  "n" is  equal  to  the  number  of 
integrations  for  that  technology 


(0,1)  scale 


Component  SRL*  represents  Technology  “X"  and  its  I  RLs  considered 


Composite  SRL 

Composite  SRL  =  1/  3  (  0.54  +  0.43  +  0.59  ) 


0.52 


The  Composite  SRL  provides  an  overall  assessment  of  the  system  readiness 

Both  individual  and  composite  scores  provide  key  insights  into  the  actual  maturity  of  the 
system  as  well  as  where  risk  may  He  and  attention  directed  for  greatest  benefit 


Sauser,  B.,  J.  Ramirez- Marquez,  D.  Henry  and  D.  DiMarzio.  (2007).  “A  System  Maturity  Index  for  the  Systems 
i neerinq  Life  Cycle.”  International  Journal  of  Industrial  and  Systems  Engineering.  3(6V  (forthcoming) _ 
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What  is  an  IRL? 

A  systematic  measurement  reflecting  the  status  of  an  integration 

connecting  two  particular  technologies 
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IRL 

Definition 

9 

Integration  is  Mission  Proven  through  successful  mission  operations. 

8 

Actual  integration  completed  and  Mission  Qualified  through  test  and  demonstration,  in  the  system 
environment. 

7 

The  integration  of  technologies  has  been  Verified  and  Validated  with  sufficient  detail  to  be  actionable. 

6 

The  integrating  technologies  can  Accept,  Translate,  and  Structure  Information  for  its  intended  application. 

5 

There  is  sufficient  Control  between  technologies  necessary  to  establish,  manage,  and  terminate  the 
integration. 

4 

There  is  sufficient  detail  in  the  Quality  and  Assurance  of  the  integration  between  technologies. 

3 

There  is  Compatibility  (i.e.  common  language)  between  technologies  to  orderly  and  efficiently  integrate  and 
interact. 

2 

There  is  some  level  of  specificity  to  characterize  the  Interaction  (i.e.  ability  to  influence)  between 
technologies  through  their  interface. 

1 

An  Interface  between  technologies  has  been  identified  with  sufficient  detail  to  allow  characterization  of  the 
relationship. 

Gove,  R.  (2007)  Development  of  an  Integration  Ontology  for  Systems  Operational  Effectiveness.  M.S.  Thesis. 
Stevens  Institute  of  Technology.  Hoboken,  NJ 


SRL  Algorithm  Sensitivity  Evaluated 

•  Observed  that  the  SRL  algorithm  did  not  take  into  account  the 
varying  levels  of  “importance”  between  technologies 

•  Examined  the  sensitivity  of  the  algorithms  to  changes  in  the  TRL 
and  IRL  ratings  of  systems  with  varying  levels  of  importance 


•  Modified  the  methodology  to  automatically  include  weightings  for 
those  technologies  that  are  most  important  by  looking  at  operational 
“strings”  or  mission  threads 


SRL  Response  Analysis 


IML  =  1  *  Indicates  unreasonable  combination  IML  =  4 


Components  to  be  integrated  are  selected  and 
interfaces  identified 


TRL 

Composite  SRL 

1 

0.06 

3 

0.17 

5 

0.28 

7 

0.39 

9 

0.51* 

IML  =  7 

End-to-end  system  integration  accomplished; 
_ prototype  demonstrated _ 


TRL 

Composite  SRL 

1 

0.10* 

3 

0.29* 

5 

0.49 

7 

0.68 

9 

0.88 

Integration  and  data  requirements  are  defined; 
low  fidelity  experimentation 


TRL 

Composite  SRL 

1 

0.08 

3 

0.23 

5 

0.38 

7 

0.54 

9 

0.69* 

IML  =  9 

System  installed  and  deployed  with  mission 
_ pmven  operation _ 


TRL 

Composite  SRL 

1 

0.11* 

3 

0.33* 

5 

0.56* 

7 

0.78 

9 

1.00 
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Algorithms  Evaluated  for  Sensitivity 


TRL  Variation  Analysis 

All  TRLs  in  the  system  are  set  to  9  with  the  exception  of  the  one 
corresponding  to  the  system  in  each  row,  which  was  set  to  1. 


IRL  Variation  Analysis 

All  IRLs  in  the  system  are  set  to  9  with  the  exception  of  the  one 
corresponding  to  the  link  in  each  row,  which  was  set  to  1 


Standard  Non-connected, 

Methodology  Self  IRLs  =  0 


Sys 

String 

Sys 

String 

MPCE 

6  Connections 

Used  by  all  Threads 

8.6 

7.9 

7.9 

7.2 

Radar 

1  Connections 

Used  by  all  Threads 

8.6 

7.9 

8.8 

8.5 

MH-60S 

7  Connections 

Used  by  5  Threads 

8.6 

8.4 

7.7 

8.1 

COBRA 

1  Connections 

Used  by  1  Thread 

8.6 

8.9 

8.8 

8.9 

NOTE:  There  are  9  total  threads 


Standard  Non-connected, 

Methodology  Self  IRLs  =  0 


Sys 

String 

Sys 

String 

MPCE -CMS 

Used  by  all  Threads 

9.0 

8.7 

8.6 

8.0 

Radar  -  CMS 

Used  by  all  Threads 

9.0 

8.7 

8.6 

8.0 

MH-60S  -  MPCE 

Used  by  5  Threads 

9.0 

8.8 

8.6 

8.4 

COBRA  -  VTUAV 

Used  by  1  Thread 

9.0 

9.0 

8.6 

8.9 

NOTE:  There  are  9  total  threads 


Comparative  Sensitivity  —  A  look  at  how  the  algorithms  penalized  the  SRL  rating  relative  to  one  another  (1  is  most  severe) 


Standard 

Methodology 

Non-connected, 
Self  IRLs  =  0 

Standard 

Methodology 

Non-connected, 
Self  IRLs  =  0 

Sys 

String 

Sys 

String 

Sys 

String 

Sys 

String 

1.)  MPCE 

1,4 

1,2 

2 

i 

1.)  MPCE  -  CMS 

1,4 

1,2 

1,4 

1,2 

2.)  MH-60S 

1,4 

3 

1 

2 

2.)  MH-60S  -  MPCE 

1,4 

3 

1,4 

3 

3.)  Radar 

1,4 

1,2 

3,4 

3 

tion 

3.)  Radar  -  CMS 

1,4 

1,2 

1,4 

1,2 

4.)  COBRA 

1,4 

4 

3,4 

4 

DS1UM 

4.)  COBRA  -  VTUAV 

1,4 

4 

1,4 

4 

